{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/ai-platform-unified/notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/ai-platform-unified/notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/ai-platform-samples/raw/master/ai-platform-unified/notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb\">\n",
        "      Open in Google Cloud Notebooks\n",
        "    </a>\n",
        "  </td>    \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2rM_9Ml7-W2"
      },
      "source": [
        "# Metrics visualization and run comparison using the KFP SDK"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "In this notebook, you will learn how to use [the Kubeflow Pipelines (KFP) SDK](https://www.kubeflow.org/docs/components/pipelines/) to build [Vertex Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines) that generate model metrics and metrics visualizations; and how to compare pipeline runs.\n",
        "\n",
        "### Objective\n",
        "\n",
        "In this example, you'll learn:\n",
        "- how to generate ROC curve and confusion matrix visualizations for classification results\n",
        "- how to write metrics \n",
        "- how to compare metrics across pipeline runs\n",
        "\n",
        "\n",
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI Training\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about pricing for [Vertex AI](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ze4-nDLfK4pw"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
        "all the requirements to run this notebook. You can skip this step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gCuSR8GkAgzl"
      },
      "source": [
        "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
        "You need the following:\n",
        "\n",
        "* The Google Cloud SDK\n",
        "* Git\n",
        "* Python 3\n",
        "* virtualenv\n",
        "* Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Google Cloud guide to [Setting up a Python development\n",
        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
        "for meeting these requirements. The following steps provide a condensed set of\n",
        "instructions:\n",
        "\n",
        "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
        "\n",
        "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
        "\n",
        "1. [Install\n",
        "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
        "   and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
        "\n",
        "1. To install Jupyter, run `pip install jupyter` on the\n",
        "command-line in a terminal shell.\n",
        "\n",
        "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
        "\n",
        "1. Open this notebook in the Jupyter Notebook Dashboard."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install additional packages\n",
        "\n",
        "Install the KFP SDK and the Vertex SDK for Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IaYsrh0Tc17L"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
        "USER_FLAG = \"\"\n",
        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    USER_FLAG = \"--user\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F9dJ5Of-dORl"
      },
      "outputs": [],
      "source": [
        "! pip install {USER_FLAG} kfp google-cloud-aiplatform matplotlib --upgrade"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hhq5zEbGg0XX"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EzrelQZ22IZj"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs\n",
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6GPgNN7eeX1l"
      },
      "source": [
        "Check the version of the package you installed.  The KFP SDK version should be >=1.6."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NN0mULkEeb84"
      },
      "outputs": [],
      "source": [
        "!python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWEdiXsJg0XY"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "This notebook does not require a GPU runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gtkz_kXrlg-g"
      },
      "source": [
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "1. [Enable the Vertex AI, Cloud Storage, and Compute Engine APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage-component.googleapis.com). \n",
        "\n",
        "1. Follow the \"**Configuring your project**\" instructions from the Vertex Pipelines documentation.\n",
        "\n",
        "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "PROJECT_ID = \"\"\n",
        "\n",
        "# Get your Google Cloud project ID from gcloud\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID: \", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJYoRfYng0XZ"
      },
      "source": [
        "Otherwise, set your project ID here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "riG_qUokg0XZ"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
        "    PROJECT_ID = \"python-docs-samples-tests\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "06571eb4063b"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "697568e92bd6"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dr--iN2kAylZ"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using Google Cloud Notebooks**, your environment is already\n",
        "authenticated. Skip this step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "**If you are using Colab**, run the cell below and follow the instructions\n",
        "when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "1. In the Cloud Console, go to the [**Create service account key**\n",
        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
        "\n",
        "2. Click **Create service account**.\n",
        "\n",
        "3. In the **Service account name** field, enter a name, and\n",
        "   click **Create**.\n",
        "\n",
        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
        "into the filter box, and select\n",
        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
        "\n",
        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
        "local environment.\n",
        "\n",
        "6. Enter the path to your service account key as the\n",
        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# If on Google Cloud Notebooks, then don't execute this code\n",
        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxhCPW6e46EF"
      },
      "source": [
        "### Create a Cloud Storage bucket as necessary\n",
        "\n",
        "You will need a Cloud Storage bucket for this example.  If you don't have one that you want to use, you can make one now.\n",
        "\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
        "Cloud Storage buckets.\n",
        "\n",
        "You may also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
        "available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). You may\n",
        "not use a Multi-Regional Storage bucket for training with Vertex AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}\n",
        "REGION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cf221059d072"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ucvCsknMCims"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vhOb7YnwClBb"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XoEqT2Y4DJmf"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YYtGjGG45ELJ"
      },
      "source": [
        "Define some constants. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5zmD19ryCre7"
      },
      "outputs": [],
      "source": [
        "PATH=%env PATH\n",
        "%env PATH={PATH}:/home/jupyter/.local/bin\n",
        "\n",
        "USER = \"your-user-name\"  # <---CHANGE THIS\n",
        "PIPELINE_ROOT = \"{}/pipeline_root/{}\".format(BUCKET_NAME, USER)\n",
        "\n",
        "PIPELINE_ROOT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IprQaSI25oSk"
      },
      "source": [
        "Do some imports:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UFDUBveR5UfJ"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "from kfp import dsl\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.dsl import ClassificationMetrics, Metrics, Output, component\n",
        "from kfp.v2.google.client import AIPlatformClient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xrwWW8iczOTN"
      },
      "source": [
        "### Initialize the Vertex SDK for Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NjBWi3-uzT71"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IbN_49SUW7b7"
      },
      "source": [
        "## Define a pipeline\n",
        "\n",
        "In this section, you define a pipeline that demonstrates some of the metrics logging and visualization features.\n",
        "\n",
        "The example pipeline has three steps.\n",
        "First define three pipeline *components*, then define a pipeline that uses them.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QW9EmIXmYLa6"
      },
      "source": [
        "### Define Pipeline components\n",
        "\n",
        "In this section, you define some Python function-based components that use scikit-learn to train some classifiers and produce evaluations that can be visualized. \n",
        "\n",
        "Note the use of the `@component()` decorator in the definitions below. You can optionally set a list of packages for the component to install; the base image to use (the default is a Python 3.7 image); and the name of a component YAML file to generate, so that the component definition can be shared and reused."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7jihcuOSx2_"
      },
      "source": [
        "The first component shows how to visualize an *ROC curve*. \n",
        "Note that the function definition includes an output called `wmetrics`, of type `Output[ClassificationMetrics]`. You can visualize the metrics in the Pipelines user interface in the Cloud Console.\n",
        "\n",
        "To do this, this example uses the artifact's `log_roc_curve()` method. This method takes as input arrays with the false positive rates, true positive rates, and thresholds, as [generated by the `sklearn.metrics.roc_curve` function](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html).\n",
        "\n",
        "When you evaluate the cell below, a task factory function called `wine_classification` is created, that is used to construct the pipeline definition.  In addition, a component YAML file is created, which can be shared and loaded via file or URL to create the same task factory function.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Gdfgm96eSx2_"
      },
      "outputs": [],
      "source": [
        "@component(\n",
        "    packages_to_install=[\"sklearn\"],\n",
        "    base_image=\"python:3.9\",\n",
        "    output_component_file=\"wine_classif_component.yaml\",\n",
        ")\n",
        "def wine_classification(wmetrics: Output[ClassificationMetrics]):\n",
        "    from sklearn.datasets import load_wine\n",
        "    from sklearn.ensemble import RandomForestClassifier\n",
        "    from sklearn.metrics import roc_curve\n",
        "    from sklearn.model_selection import cross_val_predict, train_test_split\n",
        "\n",
        "    X, y = load_wine(return_X_y=True)\n",
        "    # Binary classification problem for label 1.\n",
        "    y = y == 1\n",
        "\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
        "    rfc = RandomForestClassifier(n_estimators=10, random_state=42)\n",
        "    rfc.fit(X_train, y_train)\n",
        "    y_scores = cross_val_predict(rfc, X_train, y_train, cv=3, method=\"predict_proba\")\n",
        "    fpr, tpr, thresholds = roc_curve(\n",
        "        y_true=y_train, y_score=y_scores[:, 1], pos_label=True\n",
        "    )\n",
        "    wmetrics.log_roc_curve(fpr, tpr, thresholds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T1_SEkm9S_Fm"
      },
      "source": [
        "The second component shows how to visualize a *confusion matrix*, in this case for a model trained using `SGDClassifier`.\n",
        "\n",
        "As with the previous component, you create a `metricsc` output artifact of type `Output[ClassificationMetrics]`.  Then, use the artifact's `log_confusion_matrix` method to visualize the confusion matrix results, as generated by the [sklearn.metrics.confusion_matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bKVP6EGrS_Fm"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"sklearn\"], base_image=\"python:3.9\")\n",
        "def iris_sgdclassifier(\n",
        "    test_samples_fraction: float,\n",
        "    metricsc: Output[ClassificationMetrics],\n",
        "):\n",
        "    from sklearn import datasets, model_selection\n",
        "    from sklearn.linear_model import SGDClassifier\n",
        "    from sklearn.metrics import confusion_matrix\n",
        "\n",
        "    iris_dataset = datasets.load_iris()\n",
        "    train_x, test_x, train_y, test_y = model_selection.train_test_split(\n",
        "        iris_dataset[\"data\"],\n",
        "        iris_dataset[\"target\"],\n",
        "        test_size=test_samples_fraction,\n",
        "    )\n",
        "\n",
        "    classifier = SGDClassifier()\n",
        "    classifier.fit(train_x, train_y)\n",
        "    predictions = model_selection.cross_val_predict(classifier, train_x, train_y, cv=3)\n",
        "    metricsc.log_confusion_matrix(\n",
        "        [\"Setosa\", \"Versicolour\", \"Virginica\"],\n",
        "        confusion_matrix(\n",
        "            train_y, predictions\n",
        "        ).tolist(),  # .tolist() to convert np array to list.\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_eNs1fsTOkK2"
      },
      "source": [
        "The following component also uses the \"iris\" dataset, but trains a `LogisticRegression` model.  It logs model `accuracy` in the `metrics` output artifact."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WTlE_EwsOkK4"
      },
      "outputs": [],
      "source": [
        "@component(\n",
        "    packages_to_install=[\"sklearn\"],\n",
        "    base_image=\"python:3.9\",\n",
        ")\n",
        "def iris_logregression(\n",
        "    input_seed: int,\n",
        "    split_count: int,\n",
        "    metrics: Output[Metrics],\n",
        "):\n",
        "    from sklearn import datasets, model_selection\n",
        "    from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "    # Load digits dataset\n",
        "    iris = datasets.load_iris()\n",
        "    # # Create feature matrix\n",
        "    X = iris.data\n",
        "    # Create target vector\n",
        "    y = iris.target\n",
        "    # test size\n",
        "    test_size = 0.20\n",
        "\n",
        "    # cross-validation settings\n",
        "    kfold = model_selection.KFold(\n",
        "        n_splits=split_count, random_state=input_seed, shuffle=True\n",
        "    )\n",
        "    # Model instance\n",
        "    model = LogisticRegression()\n",
        "    scoring = \"accuracy\"\n",
        "    results = model_selection.cross_val_score(model, X, y, cv=kfold, scoring=scoring)\n",
        "    print(f\"results: {results}\")\n",
        "\n",
        "    # split data\n",
        "    X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
        "        X, y, test_size=test_size, random_state=input_seed\n",
        "    )\n",
        "    # fit model\n",
        "    model.fit(X_train, y_train)\n",
        "\n",
        "    # accuracy on test set\n",
        "    result = model.score(X_test, y_test)\n",
        "    print(f\"result: {result}\")\n",
        "    metrics.log_metric(\"accuracy\", (result * 100.0))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YTh_chI1PEpb"
      },
      "source": [
        "### Define a pipeline that uses the components\n",
        "\n",
        "Next, define a simple pipeline that uses the components that were created in the previous section."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "289jqF_XUEUO"
      },
      "outputs": [],
      "source": [
        "@dsl.pipeline(\n",
        "    # Default pipeline root. You can override it when submitting the pipeline.\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    # A name for the pipeline.\n",
        "    name=\"metrics-pipeline-v2\",\n",
        ")\n",
        "def pipeline(seed: int, splits: int):\n",
        "    wine_classification_op = wine_classification()  # noqa: F841\n",
        "    iris_logregression_op = iris_logregression(  # noqa: F841\n",
        "        input_seed=seed, split_count=splits\n",
        "    )\n",
        "    iris_sgdclassifier_op = iris_sgdclassifier(test_samples_fraction=0.3)  # noqa: F841"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Hl1iYEKSzjP"
      },
      "source": [
        "## Compile and run the pipeline\n",
        "\n",
        "Now, you're ready to compile the pipeline:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7PwUFV-MleGs"
      },
      "outputs": [],
      "source": [
        "from kfp.v2 import compiler  # noqa: F811\n",
        "\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline, package_path=\"metrics_pipeline_job.json\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qfNuzFswBB4g"
      },
      "source": [
        "The pipeline compilation generates the `metrics_pipeline_job.json` job spec file.\n",
        "\n",
        "Next, instantiate an API client object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hl5Q74_gkW2c"
      },
      "outputs": [],
      "source": [
        "from kfp.v2.google.client import AIPlatformClient  # noqa: F811\n",
        "\n",
        "api_client = AIPlatformClient(\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jrn6saiQsPh"
      },
      "source": [
        "Then, you run the defined pipeline like this: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R4Ha4FoDQpkd"
      },
      "outputs": [],
      "source": [
        "response = api_client.create_run_from_job_spec(\n",
        "    job_spec_path=\"metrics_pipeline_job.json\",\n",
        "    job_id=f\"metrics-pipeline-v2{TIMESTAMP}-1\",\n",
        "    # pipeline_root=PIPELINE_ROOT  # this argument is necessary if you did not specify PIPELINE_ROOT as part of the pipeline definition.\n",
        "    parameter_values={\"seed\": 7, \"splits\": 10},\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GvBTCP318RKs"
      },
      "source": [
        "Click on the generated link to see your run in the Cloud Console.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PSTPSsb0bIuA"
      },
      "source": [
        "## Pipeline run comparisons\n",
        "\n",
        "Next, generate another pipeline run that uses a different `seed` and `split` for the `iris_logregression` step.\n",
        "\n",
        "Submit the new pipeline run:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tYbKwyHR59MN"
      },
      "source": [
        "## Comparing pipeline runs in the UI\n",
        "\n",
        "Next, generate another pipeline run that uses a different `seed` and `split` for the `iris_logregression` step.\n",
        "\n",
        "Submit the new pipeline run:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8vglQehcbOuz"
      },
      "outputs": [],
      "source": [
        "response = api_client.create_run_from_job_spec(\n",
        "    job_spec_path=\"metrics_pipeline_job.json\",\n",
        "    job_id=f\"metrics-pipeline-v2{TIMESTAMP}-2\",\n",
        "    # pipeline_root=PIPELINE_ROOT  # this argument is necessary if you did not specify PIPELINE_ROOT as part of the pipeline definition.\n",
        "    parameter_values={\"seed\": 5, \"splits\": 7},\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EH3fF98_gZKh"
      },
      "source": [
        "When both pipeline runs have finished, compare their results by navigating to the pipeline runs list in the Cloud Console, selecting both of them, and clicking **COMPARE** at the top of the Console panel.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XLQWpvWP5lUX"
      },
      "source": [
        "## Comparing the parameters and metrics of pipeline runs from their tracked metadata\n",
        "\n",
        "In this section, you use the Vertex SDK for Python to compare the parameters and metrics of the pipeline runs. Wait until the pipeline runs have finished to run this section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OwyJWM62xC0a"
      },
      "source": [
        "### Extract metrics and parameters into a pandas dataframe for run comparison\n",
        "\n",
        "Ingest the metadata for all runs of pipelines named `metrics-pipeline-v2` into a pandas dataframe."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EKXW_8ByiCQI"
      },
      "outputs": [],
      "source": [
        "pipeline_df = aiplatform.get_pipeline_df(pipeline=\"metrics-pipeline-v2\")\n",
        "pipeline_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1fMi2wtxI9j"
      },
      "source": [
        "### Parallel coordinates plot of parameters and metrics\n",
        "\n",
        "With the metric and parameters in a dataframe, you can perform further analysis to exetract useful information. The following example compares data from each run using a parallel coordinate plot. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v_WqXRTkiCQJ"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "plt.rcParams[\"figure.figsize\"] = [15, 5]\n",
        "\n",
        "pipeline_df[\"param.input:seed\"] = pipeline_df[\"param.input:seed\"].astype(np.float16)\n",
        "pipeline_df[\"param.input:splits\"] = pipeline_df[\"param.input:splits\"].astype(np.float16)\n",
        "\n",
        "ax = pd.plotting.parallel_coordinates(\n",
        "    pipeline_df.reset_index(level=0),\n",
        "    \"run_name\",\n",
        "    cols=[\"param.input:seed\", \"param.input:splits\", \"metric.accuracy\"],\n",
        "    # color=['blue', 'green', 'pink', 'red'],\n",
        ")\n",
        "ax.set_yscale(\"symlog\")\n",
        "ax.legend(bbox_to_anchor=(1.0, 0.5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YFWIJUsSOSRe"
      },
      "source": [
        "### Plot ROC curve and calculate AUC number"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CcmCKVOKQLT-"
      },
      "source": [
        "In addition to basic metrics, you can extract complex metrics and perform further analysis using the `get_pipeline_df` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rswujUa4wpB3"
      },
      "outputs": [],
      "source": [
        "pipeline_df = aiplatform.get_pipeline_df(pipeline=\"metrics-pipeline-v2\")\n",
        "pipeline_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zqo-GgBuwyrn"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame(pipeline_df[\"metric.confidenceMetrics\"][0])\n",
        "auc = np.trapz(df[\"recall\"], df[\"falsePositiveRate\"])\n",
        "plt.plot(df[\"falsePositiveRate\"], df[\"recall\"], label=\"auc=\" + str(auc))\n",
        "plt.legend(loc=4)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6yRt6JlmrGuG"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "- delete Cloud Storage objects that were created.  Uncomment and run the command in the cell below **only if you are not using the `PIPELINE_ROOT` path for any other purpose**.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mkdPFlpwrGuP"
      },
      "outputs": [],
      "source": [
        "# Warning: this command will delete ALL Cloud Storage objects under the PIPELINE_ROOT path.\n",
        "# ! gsutil -m rm -r $PIPELINE_ROOT"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "metrics_viz_run_compare_kfp.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
